The invention relates to a method of data processing applied in a neural network structure which method comprises:
determining potentials y.sub.k of output neurons k from states x.sub.l of receiving neurons 1, the potentials and the states being interrelated through synaptic coefficients C.sub.kl ; PA1 determining quantities for obtaining increments for new synaptic coefficients C.sub.kl.sup.(n)) and calculating a norm ##EQU1## PA1 adding to a preceeding norm m.sub.k associated with previously obtained synaptic coefficient terms indicative of said related quantities. PA1 determining first terms (.delta.C.sub.kl).sup.2 for each .delta.C.sub.kl for the associated synaptic coefficients relating to output neuron k; PA1 determining second terms 2C.sub.kl .delta.C.sub.kl relating to output neuron k; PA1 adding the first terms and the second terms to the preceding norm m.sub.k. PA1 a processing unit which determines the potential y.sub.k of the receiving neuron by operating in parallel on all the input neurons 1 such that: ##EQU12## and which determines the new synaptic coefficients C.sub.kl.sup.(n) such that: ##EQU13## where .DELTA..sub.k is a variation associated with the output neuron k, a unit for storing the previous norm m.sub.k relating to the receiving neuron k, PA1 a unit for computing the norm ##EQU14## of the vector of states x.sub.l, a computational unit which determines the new norm m.sub.k.sup.(n) on the basis of the norm ##EQU15## and the increment .DELTA..sub.k, and the potential of the output neuron y.sub.k coming from the processing unit, such that: ##EQU16##
It further relates to a neuronal network structure which applies the method.
The neuronal networks find their applications in image processing, speech processing, etc.
The neuronal networks are made up of automata interconnected by synapses to which synaptic coefficients are assigned. They permit processing of problems which conventional sequential computers do with difficulty.
In order to process a given problem, the neuronal networks must first learn and to carry it out. This phase, called learning, makes use of examples.
For numerous algorithms, the results that should be obtained as output with these examples as input are generally known in advance. Initially, the neuronal network which is not yet adapted to the envisaged task will deliver erroneous results. An error E.sup.p indicative of a difference between the results obtained and those that ought to have been obtained is thus determined and, according to a modification criterion, the synaptic coefficients are modified in order to permit the neuronal network to learn the chosen example. This step is repeated over the batch of examples considered necessary for an adequate learning by the neuronal network.
A very common method for carrying out this modification, in the case of layered networks, is that of backpropagation of the gradient. To this end, the components of the gradient g.sub.j,L of the previous error E.sup.p (calculated on the last layer L) with respect to each potential y.sub.j,L of each neuron is determined. These components are then backpropagated within the neuronal network. The potentials y.sub.j,L are those quantities that are operated upon by a non-linear function, the non-linear character of this function being an essential aspect of neuronal networks in general.
This method is described, for example, in: D. E. Rumelhart, D. E. Hinton, and R. J. Williams, "Learning Internal Representation by Error Propagation", in D. E. Rumelhart, and J. L. McClelland (Eds), "Parallel Distributed Processing: Exploration in the Microstructure of Cognition", Vol. 1, Foundations, MIT Press (1986).
Backpropagation of the gradient is one of the numerous learning rules which can be put into the form: ##EQU2## where C.sub.kl and C.sub.kl.sup.(n) are the old and new synaptic coefficients, respectively, as associated with the synaps between output neuron k and input neuron 1, .DELTA..sub.k, and is a variation which depends uniquely on the receiving neuron k, x.sub.l is the state of the input neuron 1.
These rules belong to the family of Hebbian rules. Some of these rules make use of the computation of the norm m.sub.k of the synaptic vector C.sub.k whose components C.sub.kl are the synaptic coefficients associated with a single output neuron k: ##EQU3##
These rules can be applied within parallel processing neuronal network structures. In this case, the organization of such structures permits rapid processing of the data, finishing up, at the end of the learning steps, with the computation of the new synaptic coefficients.
When the new synaptic coefficients have been computed, it is then possible to determine their respective norms for application in the learning rule under consideration.
Thus, the parallel processing neuronal network structure, initially provided to speed up processing, finds itself penalized by sequential steps which necessitate accessing of the synaptic coefficients in the memory and further arithmetic operations.
The problem to be solved is therefore how to compute the norms of the new synaptic coefficients without penalizing the execution speed of the processing structures.